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CLAIMS 

What is claimed is: 

1. A compxiter-implemented method of reducing the number of hypotheses for 
locations of an active object in a saved image generated by a graphical user interface 

5 (GUI) of an application program comprising: 

determining possible triangles for the saved image, wherein vertices of the 
triangles are points where the active object and any two additional objects of the saved 
image are located; 

determining possible pairs of hypotheses corresponding to the two additional 
10 objects from a current playback image corresponding to the saved image; 

determining triangles corresponding to the possible pairs of hypotheses and 
filtering the detected triangles; 

determining vertices with weights for similar triangles of the current playback 

image; and 

15 calculating a weight for every hypothesis of the active object and filtering 

hypotheses of the active object on the current playback image based on a weight bound. 

2. The method of claim 1, wherein determining possible triangles comprises 
decreasing the number of possible triangles by using only triangles having an edge 
between two additional objects whose length is less than a predetermined distance. 

20 3. The method of claim 1, wherein determining possible pairs of hypotheses 

comprises decreasing the number of possible triangles by using only triangles which have 
an edge between two additional objects whose length is within a calculated interval based 
on a first coefficient value. 

4. The method of claim 1, wherein determining triangles corresponding to the 
25 pairs of hypotheses comprises decreasing the number of triangles by using only triangles 

which have angles within a calculated interval based on a second coefficient value. 

5. The method of claim 1, wherein a weight of a selected vertex of a triangle 
comprises 1.0 multiplied by the number of vertices of other triangles coincident at the 
selected vertex. 

30 6. The method of claim U wherein calculating a weight for every hypothesis of 

the active object comprises calculating the sum of distances between the location of the 
hypothesis and all corresponding vertices using a third coefficient value and an upper 
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boiind value. 

7. The method of claim 1, wherein filtering the hypotheses of the active object 
comprises filtering a hypothesis if the calculated weight of the hypothesis is less than the 
weight boxmd. 

5 8. The method of claim 1, further comprising identifying the best hypothesis for 

the active object based on the filtering. 

9. An article comprising: a machine accessible medivim containing mstructions, 
which when executed, result in reducing the number of hypotheses for locations of an 
active object in a saved image generated by a graphical user interface (GUI) of an 
1 0 application program by 

determining possible triangles for the saved image, wherein vertices of the 
triangles are points where the active object and any two additional objects of the saved 
image are located; 

determimng possible pairs of hypotheses corresponding to the two additional 
1 5 objects fi'om a current playback image corresponding to the saved image; 

determming triangles corresponding to the possible pairs of hypotheses and 
filtering the detected triangles; 

determining vertices with weights for similar triangles of the current playback 

image; and 

20 calculating a weight for every hypothesis of the active object and filtering 

hypotheses of the active object on the current playback image based on a weight bound. 

10. The article of claim 9, wherein instructions to determine possible triangles 
comprise instructions to decrease the number of possible triangles by using only triangles 
havmg an edge between two additional objects whose length is less than a predetermmed 

25 distance. 

11. The article of claim 9, wherein instructions to determine possible pahs of 
hypotheses comprise inst^tions to decrease the number of possible triangles by usmg 
only triangles which have an edge between two additional objects whose lengtii is withm 
a calculated interval based on a first coefficient value. 

30 12. The article of claun 9, wherein instructions to determme triangles 

corresponding to the pairs of hypotheses comprise instructions to decrease the number of 
triangles by using only triangles which have angles wilhm a calculated interval based on 
a second coefficient value. 
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13. The article of claim 9, wherein a weight of a selected vertex of a triangle 
comprises 1.0 multiplied by the number of vertices of other triangles coincident at the 
selected vertex. 

14. The article of claim 9, wherein instructions to calculate a weight for every 
5 hypothesis of the active object comprise instructions to calculate the sum of distances 

between the location of the hypothesis and all corresponding vertices using a third 
coefficient value and an upper bound value. 

15. The article of claim 9, wherein instructions to filter the hypotheses of the 
active object comprise instructions to filter a hypothesis if the calculated weight of the 

1 0 hypothesis is less than the weight bound. 

16. The article of claim 9, further comprising instructions to identify the best 
hypothesis for the active object based on the filtering. 

17. A cognitive control framework system for automatically controlling execution 
of an application program having a graphical user interface comprising: 

15 a recording component adapted to capture user input data and images displayed 

by the graphical user interface during a recording phase of execution of the application 
program, and to analyze the captured user input data and displayed images to generate an 
execution scenario during the recording phase; and 

a playback component adapted to perform image analysis on images displayed by 

20 the graphical user interface as a result of processing the simulated user input data during 
the playback phase and captured displayed images from the recording phase by, the 
playback component being adapted to reduce the number of hypotheses for locations of 
an active object in a saved image generated by a graphical user interface (GUI) of an 
application program by 

25 determming possible triangles for the saved image, wherem vertices of the 

triangles are points where the active object and any two additional objects of the 
saved image are located; 

determining possible pairs of hypotheses corresponding to the two additional 
objects from a current playback image corresponding to the saved umage; 
3Q determining triangles coirespondmg to the possible pairs of hypotheses and 

filtering the detected triangles; 

determining vertices with weights for similar triangles of the current 

playback image; and 
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calculating a weight for every hypothesis of the active object and filtering 
hypotheses of the active object on the current playback image based on a weight 
bound. 

18. The system of claim 17, wherein determining possible triangles comprises 
5 decreasing the number of possible triangles by using only triangles having an edge 

between two additional objects whose lengtii is less than a predetermined distance. 

19. The system of claim 17, wherein determining possible pairs of hypotheses 
comprises decreasing the number of possible triangles by using only triangles which have 
an edge between two additional objects whose length is within a calculated interval based 

10 on a first coefficient value, 

20. The system of claim 17, wherein determining triangles corresponding to the 
pairs of hypotheses comprises decreasing the number of triangles by using only triangles 
which have angles within a calculated mterval based on a second coefficient value. 

21. The system of claim 17, wherein a weight of a selected vertex of a triangle 
15 comprises 1.0 multiplied by the number of vertices of other triangles coincident at the 

selected vertex. 

22. The system of claim 17, wherein calculating a weight for every hypothesis of 
the active object comprises calculating the sum of distances between the location of the 
hypothesis and all corresponding vertices using a third coefficient value and an upper 

20 bo\md value. 

23. The system of claim 17, wherein filteriaig the hypotheses of the active object 
comprises filtering a hypothesis if the calculated weight of the hypothesis is less than the 
weight bound. 

24. The system of claim 17, wherem the playback component is adapted to 
25 identify fee best hypothesis for the active object based on the filtermg. 



